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This  project  examines  the  learning  of  complex  topics.  The  research 
was  aimed  at  two  different  aspects  of  the  study  of  learning  and  instruc- 
tion: 1)  theoretical  understanding  of  what  is  going  on  in  the  learner's 
mind;  2)  development  of  intelligent  interactive  computer-based  instruc- 
tional systems.  Two  computer-based  tutorial  systems,  Coach  and 
Instruct,  were  developed.  The  findings  of  our  research  are  summarized.^ 
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This  is  the  final  report  for  Contract  N-0001 4-76-C-0628, 
covering  the  period  of  1 February  1976  through  31  January  1979. 
The  goal  of  the  project  was  to  increase  knowledge  about  the 
psychological  mechanisms  of  learning  and  of  teaching  complex 
materials.  One  focus  of  tne  work  was  the  investigation  of  funda- 
mental principles  of  instruction;  the  other  major  focus  was  the 
development  of  operational  computer-based  tutorial  systems  which 
could  be  used  as  testbeds  for  the  study  of  these  principles,  and 
also  a3  prototypes  for  potential  applications. 

Several  computer-based  tutorial  systems  were  explored.  One 
such  system  was  the  FLOW  Tutor,  a completely  automated  tutorial 
system  for  teaching  a basic  computer  programming  language.  This 
system  uses  an  active  semantic  memory  representation  of  the 
knowledge  about  programming,  of  the  manual  being  read  by  the  stu- 
dent, and  of  the  knowledge  that  the  student  has  acquired.  The 
system  has  been  implemented  on  a computer  allowing  for  assessment 
of  its  potential  capabilities. 
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A second  tutorial  system  is  the  "incremental  tutor"  system. 
This  is  a semi-automated  facility  which  uses  a state-dependent 
teaching  strategy,  allowing  for  intelligent  interaction  between 
the  automated  parts  of  the  tutor,  a human  tutor  who  is  called  in 
when  necessary,  and  the  student.  This  system  was  also  imple- 
mented on  a computer  and  used  to  teach  students  in  experiments  on 
different  forms  of  instructional  strategies. 

These  experiments  with  tutorial  systems  and  with  the  presen- 
tation of  materials  to  students  have  provided  us  with  means  for 
the  assessment  of  teaching  strategies  and  advanced  our  theoreti- 
cal understanding  of  the  process  of  learning. 

Theoretical  work  concentrated  on  two  aspects  of  learning: 
first,  that  learning  is  not  a homogeneous  process,  but  that 
several  different  forms  of  activity  are  involved;  second,  that 
the  role  of  prior  knowledge  is  critical  and  students  must  be 
presented  with  appropriate  conceptualizations  and  metaphors  for 
the  proper  understanding  of  the  new  knowledge  they  are  acquiring. 

At  the  conclusion  of  this  contract,  two  major  experimental 
systems  for  instruction  were  implemented  on  a small  laboratory 
computer,  numerous  students  had  been  examined  going  through  the 
learning  of  several  different  topic  matters,  and  considerable 
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advance  in  our  theoretical  understanding  of  the  nature  of 
instruction  had  taken  place. 
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Overview 

This  project  examines  complex  learning.  Recent  advances  in 
our  understanding  of  the  human  memory  system,  especially  the  way 
in  which  information  is  represented,  have  led  to  significant 
advances  in  our  understanding  of  memory  and,  therefore,  of  learn- 
ing. Similarly,  recent  advances  in  the  development  of  interac- 
tive computer  systems  that  couple  with  a rich  knowledge  base  of 
information  promise  significant  advances  in  the  ability  to  devise 
interactive  instructional  systems.  Learning,  however,  involves 
more  than  the  simple  placement  of  new  information  into  a 
student's  memory.  Instruction  involves  more  than  merely  noting 
student  difficulties  and  presenting  appropriate  new  advice  or 
information. 

The  studies  reported  here  were  directed  at  two  different 
aspects  of  the  study  of  learning  and  instruction:  the  theoretical 
understanding  of  what  goes  on  in  the  mind  of  the  learner,  and  the 
development  of  intelligent  interactive  computer  systems  for 
instruction.  Significant  progress  has  been  made  in  all  domains. 
However,  considerable  work  remains  before  we  will  achieve  an  ade- 
quate understanding  of  the  learning  process.  Similarly,  consid- 
erable work  and  development  must  take  place  before  we  come  to 
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have  sophisticated,  intelligent  teaching  systems. 

The  results  already  obtained  can  have  practical  applica- 
tions. We  believe  we  have  learned  enough  about  the  learning  pro- 
cess that  advantage  can  be  taken  in  practical  teaching  and 
instructional  situations.  Small  interactive  instructional  sys- 
tems can  be  built  for  specialized  topic  domains.  Our  demonstra- 
tion project  of  an  interactive  tutor  shows  that  it  is  indeed  pos- 
sible to  have  intelligent  tutoring  of  a specialized  topic.  Our 
work  and  the  work  of  others  has  shown  that  if  the  topic  to  be 
instructed  is  chosen  with  care,  so  that  the  topic  is  properly 
constrained,  well  understood,  and  defined  with  some  care,  then 
impressive  results  can  be  obtained  with  interactive  computer- 
based  tutorial  systems  (see  Burton  & Brown,  1979!  Goldstein, 
1979). 

This  report  summarizes  the  accomplishments  of  this  project. 
The  details  have  been  presented  in  technical  reports  and  publica- 
tions, as  referenced  within  this  document.  In  addition,  a book 
summarizing  the  theoretical  aspects  of  this  project  is  in  pro- 
gress (Norman,  in  progress).  A good  brief  summary  of  the  overall 
accomplishments  was  published  in  the  Naval  Research  Reviews  (Nor- 
man & Gentner,  197b). 


I 


D.  A.  Norman 
June  9,  1979 


ONR  Final  Report 

6 


Advances  la  £hg.  Understanding  $f  Instructional  Theory 

The  basic  notion  underlying  the  development  of  instructional 
systems  is  simple.  Considerable  progress  has  been  made  in  the 
representation  of  knowledge  of  complex  topics.  These  representa- 
tions take  on  various  forms,  but  they  mostly  fall  within  the  gen- 
eric class  of  network  or  schema  theories  of  representation. 
Basically,  one  represents  the  knowledge  that  is  meant  to  be  con- 
veyed to  the  student  by  means  of  a semantic  network.  A second 
semantic  network  is  used  to  provide  a model  of  the  student,  show- 
ing the  presence  and  absence  of  relevant  knowledge  structures  for 
the  topic  to  be  acquired.  Instruction,  then,  can  be  thought  of 
as  the  process  of  transferring  from  teacher  to  student  those 
knowledge  structures  in  the  topic  matter  that  are  not  present  in 
the  student.  We  believe  this  formulation  is  a reasonably  viable 
model  for  some  kinds  of  learning  situations.  However,  it  fails 
for  lack  of  consideration  of  the  activities  engaged  in  by  the 
learner.  The  learner  is  not  a passive  being.  A learner  selects 
which  aspect  of  the  material  to  attend  to  and  then  reinterprets 
that  material,  so  that  the  information  the  instructors  think  they 
are  presenting  is  often  quite  different  from  the  information  the 
students  might  perceive  the  instructors  to  be  presenting. 
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In  our  studies  of  learning,  we  spent  considerable  effort 
examining  the  growing  knowledge  structures  of  the  students  as 
they  went  through  various  instructional  materials.  We  carefully 
devised  instructional  materials  of  various  sorts  and  on  different 
topics.  We  presented  these  to  students  in  controlled  cir- 
cumstances, sometimes  one  sentence  at  a time,  while  we  carefully 
observed  the  student's  performance. 

In  order  to  understand  even  simple  material,  considerable 
knowledge  is  required.  The  terms  and  the  concepts  that  must  be 
understood  have  a surprising  richness  of  structure.  In  order  to 
understand  a computer  text  editor,  one  has  to  understand  the  pur- 
pose of  editing,  the  nature  of  manuscripts,  and  the  way  in  which 
one  might  wish  to  alter  an  existing  manuscript.  One  must  also 
come  to  understand  computers,  at  least  at  a level  at  which  the 
operation  of  particular  programs  can  be  understood.  One  also  has 
to  know  something  of  the  file  structure  for  storage,  including 
storage  within  a particular  program,  working  storages,  and  the 
difference  between  the  immediate  RAM  memory  of  the  machine  and 
the  back-up  store  (disk,  tape).  Students  often  have  images  of 
computers  that  include  such  notions  as  omnipotence.  They  have  to 
understand  the  concept  of  a text  editor  as  a limited 
instruction-following  system  that  was  designed  by  someone  for  a 
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limited  purpose.  Thus,  text  editors  do  not  necessarily  follow 
the  organization  expected  by  the  students,  and  many  of  the  deci- 
sions involved  in  the  design  of  a text  editor  are  of  necessity 
quite  arbitrary. 

Our  goal  is  to  teach  the  computer  text  editor  to  people  who 
know  nothing  about  computers.  Indeed,  the  designers  of  these 
editing  systems  do  not  believe  it  should  be  necessary  to  under- 
stand computers  in  order  to  use  the  system.  We  soon  discovered 
that  students  were  devising  models  of  the  computer  and  of  the 
system  they  were  learning.  Their  models  tended  to  be  capricious 
and  misleading.  We  discovered  that  during  instruction,  while 
students  are  actively  interpreting  the  material  being  presented 
to  them,  it  is  essential  for  the  instructor  to  have  control  over 
the  students'  models.  This  means  that  it  is  the  instructor  who 
must  present  models  for  students  to  use . Of  course , this  poses 
severe  practical  difficulties,  since  in  order  to  present  any  body 
of  knowledge,  one  has  to  present  an  appropriate  conceptual  basis. 
As  that  basis  is  itself  a topic  matter  that  must  be  instructed, 
it  too  requires  some  conceptual  basis.  The  recursion  suggested 
by  this  analysis  is  obviously  unworkable.  One  effective  instruc- 
tional method  is  to  identify  possible  prototype  systems  that  stu- 
dents will  be  able  to  use  as  a basis,  systems  with  which  the  stu- 
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dents  can  be  expected  to  already  be  familiar.  Thus,  in  teaching 
about  the  computer  text  editor,  we  began  to  use  the  technique 
known  as  "teaching  by  analogy."  We  selected  from  the  three 
models  of  a tape  recorder,  a human  secretary,  and  a card  file  to 
illustrate  various  components  of  the  operation  of  the  text  edi- 
tor. 


Learning  by  analogy,  or  learning  by  using  metaphoric 
analysis  of  the  situation  with  reference  to  another  previously 
understood  situation  promises  to  be  an  important  area  for  future 
directions  of  this  research.  Unfortunately,  the  full  apprecia- 
tion of  the  nature  of  the  active  strategies  being  employed  by  the 
students  did  not  become  apparent  before  the  end  of  this  contract 
period,  and  so  only  the  briefest  excursion  into  teaching  via 
metaphor  was  carried  out.  We  recommend  further  studies  along 
these  lines,  believing  that  reasonably  large  increments  in  gen- 
eral knowledge  of  the  instructional  process  can  be  made  by  fol- 
lowing this  approach.  The  best  summaries  of  this  work  can  be 
found  in  Norman  (in  press)  and  in  Bott  (1979). 
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Studies  of  Memory 

Because  memory  is  a major  component  of  learning,  consider- 
able effort  was  spent  on  increasing  our  knowledge  of  the  memory 
structure.  One  set  of  studies  expanded  our  knowledge  of  the 
organizational  structure  of  memory  — the  notion  of  schemas. 
Thus,  it  is  now  believed  that  a considerable  amount  of  informa- 
tion in  memory  is  organized  into  structured  units.  Consideration 
of  schemas  leads  in  several  directions.  One  is  the  understanding 
of  the  relationship  between  the  development  and  completion  of 
schemas  and  learning.  This  led  us  to  the  realization  that  learn- 
ing can  take  place  in  at  least  three  different  modes:  accretion, 
restructuring,  and  tuning  (Rumelhart  & Norman,  197 B.)  Learning  is 
not  a unitary  process,  and  so  we  should  not  expect  a single 
explanation  to  account  for  the  different  processes  that  must  go 
on.  Similarly,  instruction  for  the  different  phases  of  learning 
must  take  on  different  forms. 

A second  aspect  of  memory  involves  the  retrieval  of  informa- 
tion previously  learned.  Norman  and  Bobrow  (1979)  pushed  the 
3tudy  of  the  retrieval  effort,  inventing  the  notion  of  "descrip- 
tion" and  its  role  in  the  process  of  retrieval.  The  use  of 
description  emphasizes  the  way  that  partial  knowledge  can  lead 
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eventually  to  appropriate  recall  in  some  circumstances,  or  to 
inappropriate  recall  or  complete  failure  in  others.  In  a related 
study,  Williams  (1978)  studied  in  detail  the  processes  by  which 
people  retrieve  memories  from  the  long  past.  The  studies  of 
schemas,  descriptions,  and  retrieval  expand  our  knowledge  of  the 
memory  system  in  ways  directly  relevant  to  the  study  of  instruc- 
tion. In  another  related  study,  Miyake  and  Norman  (in  press) 
examined  how  people  are  able  to  realize  when  they  lack  the 
appropriate  information  and  how  they  then  formulate  questions 
about  that  information.  This  is  an  issue  that  gets  at  the  heart 
of  the  interactive  process  between  the  student  and  the  instruc- 
tor, and  we  showed  that  students  ..sk  questions  primarily  when 
their  lack  of  knowledge  is  well-matched  to  the  level  of  instruc- 
tion. Students  who  are  exposed  to  instruction  at  a level  that  is 
cither  too  advanced  or  too  elementary  tend  not  to  ask  questions. 
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Computer  Based  Tutorial  Systems 

Two  major  computer-based  systems  were  developed.  One, 
called  Coach,  wa3  a fully  automated  tutorial  system  for  instruc- 
tion in  a computer  language,  based  upon  a semantic  network 
representation  of  the  topic,  of  the  instruction  manual,  and  of 
the  student's  knowledge.  The  other,  called  Instruct,  was  a 
testbed  for  instructional  research,  with  a semi-automated 
tutorial  environment,  one  that  allowed  for  incremental  increase 
in  tne  interaction  between  the  computer  system  and  the  student. 
This  system  was  used  to  explore  the  teaching  of  a variety  of 
topics . 

Coach 


Coach  is  a working  demonstration  system  capable  of  tutorial 
assistance  in  the  learning  of  a simple  computer  programming 
language . Coach  uses  a semantic  network  schema-based  knowledge 
structure  of  the  topics  that  are  being  learned,  coupled  with  an 
analysis  of  the  student's  progress  through  the  course  material  to 
interpret  the  student's  actions  and  give  relevant  coaching  advice 
where  required.  Coach  represents  a prototype  of  a new  generation 
of  computer-assisted  tutorial  methods  which  teach  via  a deep 
understanding  of  both  the  topic  matter  to  be  covered  and  the 
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knowledge  levels  of  the  students.  As  might  be  expected,  however, 
the  system  requires  considerable  power  and  memory  capacity.  It 
is  not  yet  feasible  to  use  this  system  in  a real  environment  with 
the  machines  that  are  realistically  available  today.  But  this 
situation  is  rapidly  changing,  and  it  is  likely  that  sophisti- 
cated tutorial  systems  along  the  lines  of  Coach  could  be  imple- 
mented on  a machine  costing  in  the  $10-20,000  range  within  the 
next  few  years,  with  prices  continuing  to  drop  at  a rapid  rate  as 
the  technology  required  to  build  machines  of  the  LISP  class 
expands . 

Coach  has  not  been  used  to  tutor  real  students.  On  the 
machines  currently  available  to  us,  we  are  unable  to  maintain  a 
system  capable  of  teaching  the  entire  topic  matter  of  the  pro- 
gramming language,  and  even  for  the  small  segment  of  material 
that  is  operating,  there  are  often  considerable  machine  delays. 
Nonetheless,  the  working  version  of  Coach  does  represent  a work- 
ing example  of  a prototype  development  in  a new  direction.  Fo  1 a 
summary  of  the  operation  of  the  Coach  system,  see  Gentner  (in 
press);  Gentner  & Norman  (1977). 


Y 
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Instruct 

A second  major  program  was  the  development  of  an  "incremen- 
tal tutor,"  an  experimental  design  for  an  interactive  system 
which  uses  the  best  of  both  human  and  automated  tutoring.  The 
aim  here  is  to  make  a system  that  will  supplement  the  human 
tutor.  This  could  enable  either  one  human  tutor  to  cover  many 
more  students  than  would  normally  be  possible,  or  it  would  allow 
relatively  unskilled  tutors  to  answer  questions.  The  basic  prem- 
ise behind  this  system  is  that  a complete  automated  tutorial  sys- 
tem for  various  complex  topics  is  beyond  the  current  state  of  the 
art.  However,  some  of  the  basic  principles  of  tutorial  instruc- 
tion can  be  implemented,  and  it  is  certainly  possible  to  devise 
intelligent  systems  that  have  some  understanding  of  the  topic 
matter  that  is  being  presented,  that  are  able  to  devise  a simpli- 
fied model  of  the  student,  and  that  can  handle  a limited  range  of 
tutorial  assistance.  Thus,  it  is  possible  to  devise  systems  that 
can  give  limited  tutorial  assistance  or  that  can  give  aid  to  a 
human  tutor  when  necessary.  That  is,  when  the  system  cannot  ful- 
fill the  task  itself,  it  can  call  in  the  human  tutor,  alert  the 
tutor  to  the  problem,  and  give  a tentative  diagnosis  of  what  is 
occurring.  It  is  even  possible  to  give  a tentative  list  of  sug- 
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ge3ted  tutorial  strategies,  letting  the  human  tutor  select  one  of 
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them  or  any  other  means  to  aid  the  student. 

The  Instruct  system  was  successful  in  its  use  as  a test-bed 
of  instructional  systems  (Norman  4 Gentner,  1978).  Some  of  the 
principles  resulting  from  these  studies  can  be  applied  to 
instructional  systems,  and  some  have  played  critical  roles  in  the 
developing  theoretical  understanding  of  learning  and  teaching 
described  in  the  first  section  of  this  report. 
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